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Speoiiii 


During  the  past  7  months,  the  research  efforts  supported  by  AFOSR  Grant 
AFOSR-84-0381  have  concentrated  on  robust  estimation  techniques  for  autoregressive 
models  and  some  related  system  theoretic  problems  associated  with  parameter  estimation 
problems  for  time  series  models.  The  motivation  for  this  work  arises  from  applications 
in  signal  processing  such  as  linear  predictive  signal  modeling,  signal  detection,  and  spec¬ 
tral  analysis.  The  overall  goal  of  this  research  has  been  to  put  together  ideas  and  tech¬ 
niques  from  statistics,  signal  processing,  and  system  theory  to  bring  new  perspectives  to 
such  problems.  This  section  will  be  devoted  to  some  discussion  of  the  technical  results 
obtained  to  date. 

Our  research  on  autoregressive  modeling  has  been  motivated  by  the  need  to  relax 
some  of  the  unrealistic  assumptions  made  by  previous  researchers,  allowing  for  robust 
estimation  in  the  presence  of  outliers  in  the  observed  signal.  We  have  made  extensive 
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use  of  a  particular  model  structure,  the  so-called  lattice  structure,  to  reduce  the  compu¬ 
tational  complexity  of  the  iterated,  reweighted  least  squares  approach  to  outlier  suppres¬ 
sion  [l].  The  structure  uses  partial  autocorrelations  to  parametrize  the  autoregression, 
resulting  in  a  decoupling  of  the  least  squares  problems.  Results  in  [1|  show  that  just  one 
or  two  iterations  of  the  basic  “data  cleaning”  procedure  provides  estimates  with  much 
better  performance  than  least  squares. 

Given  the  success  of  this  approach  to  robust  autoregressive  model  fitting,  we  have 
begun  to  explore  how  our  approach  may  be  extended  to  more  general  signal  modeling 
and  analysis  problems.  As  a  first  step,  we  examined  how  the  lattice  structure  may  be 
used  to  extract  estimates  of  the  AR  part  of  an  ARMA  model  [2].  Such  a  problem  arises 
when  applying  linear  prediction  (autoregressive  modeling)  for  spectral  estimation  of  sine 
waves  in  additive  white  noise.  We  have  shown  that  it  is  not  necessary  to  solve  a 
sequence  of  least  squares  problems  (of  order  greater  than  or  equal  to  the  autoregressive 
order)  to  get  consistent  estimates;  rather  an  extended  lattice  filter  may  be  transformed 
to  give  the  estimates  from  the  solution  of  a  single  set  of  linear  equations  (of  order  equal 
to  the  moving  average  order)  [2].  This  result  is  also  shown  to  be  equivalent  to  the 
extended  Yule- Walker  approach.  The  principal  advantage  will  come  when  we  apply  the 
robust  estimation  techniques  already  mentioned  to  this  more  general  estimation  problem. 

We  turn  now  to  a  second  area  of  research,  starting  with  some  notation  for  the  kind 
of  problems  studied.  Let  (f/],  ■  •  )  be  a  sequence  of  observed  random  variables 

whose  probability  distributions  are  described  by  a  parametrized  family  of  density  func¬ 
tions  {p*(«i,  0)}.  We  assume  that  there  exists  a  sequence  of  sufficient  statistics 

for  9,  (T i(U i),  T <^U x,U ■  ■  •  ),  so  that  all  of  the  information  about  the  parameter  6 
available  in  the  observations  up  to  instant  k  is  preserved  in  the  (fixed  dimension!)  statis¬ 
tic  Th .  This  formulation  is  applicable  to  many  parameter  estimation  problems  arising  in 
applications,  including  linear  predictive  (i.e.  autoregressive)  modeling  of  signals. 

In  the  study  of  problems  where  the  observation  record  increases,  it  is  a  natural  sys¬ 
tem  theoretic  view  to  regard  a  sequence  of  sufficient  statistics  as  defining  the 

input/output  map  of  a  dynamical  system.  In  work  completed  during  the  current  grant  _ 
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period,  done  in  collaboration  with  Eduardo  Sontag  of  Rutgers  University,  we  have 
developed  some  realization  theory  for  nonlinear,  time  varying  systems  that  can  be 
applied  to  the  study  of  state  space  models  for  sufficient  sequences  [3].  The  importance 
of  state  space  models  should  be  clear:  they  provide  a  recursive  means  of  updating  the 
sequence  of  sufficient  statistics  as  more  observations  are  made.  Actually,  for  this  to  be 
the  case,  it  is  necessary  that  the  state  space  models  be  finite  dimensional.  Results  in  [3] 
indicate  classes  of  systems  for  which  finite  dimensional  realizations  may  be  obtained  and 
give  a  criterion  for  verifying  that  a  (nonlinear,  time  varying)  state  space  model  is  a 
minimum  dimension  realization  of  its  input/output  map.  The  latter  is  an  appropriately 
formulated  theorem  of  the  usual  kind:  controllability  and  observability  of  a  realization 
imply  that  the  realization  is  minimum  dimension. 

In  [3]  we  adopted  the  framework  of  nonlinear  filtering  theory,  i.e.  the  Bayesian  per¬ 
spective,  and  proved  that  3p  -t-1  is  the  minimum  dimension  realization  of  the  recursive 
Bayes  estimator  for  the  parameters  of  a  order  autoregressive  process.  This  estimator 
is  based  on  reproducing  (or  conjugate)  families  of  densities,  but  it  is  equivalent,  at  least 
for  long  observation  records,  to  the  usual  least  squares  estimates  used  in  linear  predictive 
modeling. 

Another  subject  considered  in  [3]  is  the  relation  of  the  notion  of  realizability  of  a 
sufficient  sequence  with  other  properties  introduced  in  the  statistics  literature,  especially 
transitivity.  We  show  that  realizability  has  some  important  practical  implications 
through  its  connection  with  (finite  dimensional)  recursive  equations  for  updating  the 
sequence  of  statistics  that  are  not  necessarily  associated  with  transitivity. 

We  have  followed  up  our  work  on  realizability  with  some  further  study  of  its  impli¬ 
cations  and  limitations.  The  theory  in  (3|  requires  that  realizations  take  the  form 

X,^y  =  a,(X,,U,) 
n  =VkiX,,U,) 

where  the  functions  a  and  ri  are  differentiable,  since  we  need  some  kind  of  smoothness 
condition  to  make  sense  out  of  the  notion  of  dimension.  In  [3]  we  gave  the  following 
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example  to  show  that  there  are  sufficient  sequences  admitting  no  smooth  finite  dimen¬ 
sional  realization.  For  observations  from  a  stationary  Gaussian  process  with  unknown 
mean  and  known,  nonrational  power  spectral  density  function,  there  is  a  one-dimensional 
sufficient  sequence  defining  a  linear  input/output  map  that  has  no  smooth  finite  dimen¬ 
sional  realization,  linear  or  nonlinear.  Viewed  from  the  perspective  of  the  observed  pro¬ 
cess,  this  result  is  perhaps  not  unexpected,  since  there  is  no  smooth  finite  dimensional 
model  which  generates  such  observations. 

In  following  up  on  this  example,  we  have  now  shown  [4)  that  its  essential  feature, 
no  finite  dimensional  model  for  the  observation  process,  must  underlie  any  example  of 
this  sort.  Specifically,  a  realizability  assumption,  in  conjunction  with  existence  of  a 
sufficient  sequence,  implies  that  the  observation  process  is  equivalent  to  one  that  is  gen¬ 
erated  by  a  finite  dimensional  dynamical  system  driven  by  a  sequence  of  independent 
random  variables. 

In  further  work,  still  in  progress  but  reported  in  summary  form  in  [5],  we  have 
extended  our  approach  to  deal  with  the  problem  of  prediction  (where  the  parameters  of 
the  joint  probability  density  function  of  the  observations  are  of  no  interest).  In  practice, 
prediction  problems  usually  arise  in  connection  with  a  growing  observation  record 
instead  of  a  fixed  one.  As  described  above,  if  there  exists  a  sequence  of  sufficient  statis¬ 
tics  for  the  parameter,  as  in  the  case  of  an  autoregressive  process,  then  the  sequence  may 
be  viewed  as  the  input-output  map  of  a  dynamical  system  and  the  question  of  finite 
dimensional  realizability  may  be  investigated.  For  prediction  problems,  there  are 
interesting  relationships  between  realizability  and  a  property  called  total  sufficiency.  For 
autoregressive  processes.  For  autoregressive  processes,  this  property  holds  and  provides 
one  way  of  obtaining  parameter-independent  predictions. 
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Research  Plans 

We  turn  now  to  a  brief  overview  of  research  planned  for  the  second  year  of  the 
grant  period.  As  indicated  above,  our  research  in  robust  signal  modeling  is  ongoing, 
with  current  efforts  directed  along  two  lines.  First,  extending  the  robust  .\R  estimation 
method  to  more  general  models  where  the  lattice  filter  structure  can  be  exploited  will  be 
considered  in  detail.  Second,  we  hope  to  merge  the  two  lines  of  research  described  above 
through  the  work  on  sufficiency  and  prediction.  Specifically,  using  nominal  forms  for  the 
predictors  obtained  for  certain  parametric  models  (such  as  autoregressions)  we  plan  to 
explore  procedures  for  making  these  predictors  robust. 

There  are  several  topics  associated  with  realizability  of  sufficient  sequences  that 
remain  for  further  investigation.  We  plan  to  explore  the  implications  of  realizability  of  a 
sufficient  sequence  on  the  structure  of  the  family  of  joint  density  functions  for  the 
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observed  process.  It  appears  that  our  work  should  lead  to  a  significant  generalization  of 
the  well-known  result  for  sequences  of  independent  observations,  namely  that  realizabil¬ 
ity  implies  an  exponential  family  form.  The  realizability  assumption  appears  to  be  just 
the  right  one  to  treat  sequences  of  dependent  (correlated)  random  variables,  which  is  the 
case  of  interest  for  signal  processing  applications. 

Related  to  this  work  will  be  an  investigation  of  how  results  on  parametrized  fami¬ 
lies  of  systems  might  be  used  to  provide  a  useful  definition  of  a  parametrized  family  of 
joint  density  functions  corresponding  to  a  class  of  processes  with  “finite  dimensional 
representations.”  What  we  have  in  mind  here  is  a  definition  that  will  work  when 
sufficiency  doesn’t  help.  For  example,  a  filtered  Gaussian  white  noise  sequence  has  a 
finite  dimensional  representation  whenever  the  filter  transfer  function  is  a  rational  func¬ 
tion  (as  a  z-transform),  even  though  a  sufficient  statistic  exists  (and  by  the  work 
described  earlier,  leads  to  a  finite  dimensional  representation)  only  if  the  filter  is  all-pole. 
Removing  the  assumption  of  regularity  of  the  family  of  joint  density  functions  would 
also  be  of  interest,  allowing  a  wider  class  of  models  to  be  treated. 

Further  investigation  of  the  connections  between  sufficiency  and  prediction  for  time 
series  models  will  be  carried  out.  As  noted  above,  we  hope  to  be  able  to  use  this 
approach  as  part  of  our  work  on  robustness.  We  want  to  give  u  thorough  answer  to  the 
question  of  why  finite  dimensional  sufficient  statistics  and  finite  dimensional  predictor 
spaces  seem  to  go  hand  in  hand.  Because  there  is  some  well-developed  theory  of  approx¬ 
imate  (i.e.  in  the  sense  of  least  squares)  prediction,  we  believe  that  the  connections 
between  prediction  and  sufficiency  will  lead  to  a  reasonable  theory  of  approximation  in 
the  context  of  sufficiency,  applicable  to  signal  analysis  problems  such  as  linear  predictive 
modeling.  Clearly,  the  whole  question  of  approximation  is  an  important  one,  being  the 
essential  part  of  the  modeling  problem  in  any  real  application. 


